Skip to content

fix: flaky tests for utf8 warning test#2597

Closed
SkArchon wants to merge 8 commits intomainfrom
milinda/fix-flaky-tests
Closed

fix: flaky tests for utf8 warning test#2597
SkArchon wants to merge 8 commits intomainfrom
milinda/fix-flaky-tests

Conversation

@SkArchon
Copy link
Copy Markdown
Contributor

@SkArchon SkArchon commented Mar 6, 2026

This PR fixes a flaky test because parallel tests overrides the error handler, which means that the utf8 warning message test's error handler with the particular log observer instance might be overridden

Summary by CodeRabbit

  • Tests
    • Extracted a flaky UTF‑8 export error logs check into its own standalone test to improve isolation and stability.
    • Reordered and adjusted test flow to reduce concurrency flakes and make log assertions more reliable.
  • Bug Fixes
    • Made trace export error handling instance-local to reduce noisy/racy error reporting during test runs and improve diagnostic reliability.

Checklist

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Mar 6, 2026

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

Moved a flaky invalid-UTF8 test into its own t.Run and changed its control flow; added a syncSpanProcessor to route span export errors to an instance-local handler and replaced memory exporter registration to use that processor for per-instance error handling. No public API changes.

Changes

Cohort / File(s) Summary
Test updates
router-tests/telemetry/attribute_processor_test.go
Extracted the "invalid UTF-8 export error logs config hint" subtest into its own t.Run. The moved test now uses MakeGraphQLRequest (explicit error check) and polls logs for the sanitize_utf8 message rather than relying on response status.
Tracing setup
router/pkg/trace/meter.go
Replaced registration that used sdktrace.WithSyncer(config.MemoryExporter) with creation/registration of a custom syncSpanProcessor that forwards export errors to an instance-specific errHandler(config) instead of the global OTEL error handler.
New local span processor
router/pkg/trace/syncspan_test_exporter.go
Added syncSpanProcessor type implementing a synchronous span processor that exports sampled spans and routes export errors to a provided local error handler. No exported API changes.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed The title 'fix: flaky tests for utf8 warning test' accurately describes the main objective of the changeset—fixing flaky tests related to UTF-8 warning handling. The changes reorganize test flow, improve error handling with a custom span processor, and stabilize the UTF-8 validation test through better log polling, all directly supporting this goal.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Mar 6, 2026

Router image scan passed

✅ No security vulnerabilities found in image:

ghcr.io/wundergraph/cosmo/router:sha-9e3b0f2cd08e7cb3bb3393f9613049b5af086aab

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@router-tests/telemetry/attribute_processor_test.go`:
- Around line 198-201: Restore the eventual wait around the post-request
assertion to avoid races: after calling
xEnv.MakeGraphQLRequestOK(testenv.GraphQLRequest{...}) wrap the check that
res.Response.StatusCode equals 200 in require.Eventually (referencing
require.Eventually and res.Response.StatusCode) so the test polls until the
export/log side-effect has been observed instead of asserting immediately;
ensure the timeout and polling interval are reasonable for the async export/log
delivery.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: decaa42c-56f1-4ff5-8296-ae63562876d1

📥 Commits

Reviewing files that changed from the base of the PR and between 9ea1388 and 9736954.

📒 Files selected for processing (1)
  • router-tests/telemetry/attribute_processor_test.go

Comment thread router-tests/telemetry/attribute_processor_test.go Outdated
@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 6, 2026

Codecov Report

❌ Patch coverage is 71.42857% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.61%. Comparing base (3e28449) to head (c2ce70e).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
router/pkg/trace/syncspan_test_exporter.go 60.00% 3 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2597      +/-   ##
==========================================
- Coverage   64.11%   61.61%   -2.51%     
==========================================
  Files         301      245      -56     
  Lines       42642    25770   -16872     
  Branches     4538        0    -4538     
==========================================
- Hits        27342    15877   -11465     
+ Misses      15279     8508    -6771     
- Partials       21     1385    +1364     
Files with missing lines Coverage Δ
router/pkg/trace/meter.go 43.44% <100.00%> (ø)
router/pkg/trace/syncspan_test_exporter.go 60.00% <60.00%> (ø)

... and 544 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
router-tests/telemetry/attribute_processor_test.go (1)

240-250: Stabilize this flaky assertion by separating trigger from polling.

Line 241 currently sends a new request on every Eventually poll, and line 250 depends on logs[0] ordering. This can still be nondeterministic. Send one request, then poll only for the expected message across all matching log entries.

Suggested refactor
 	}, func(t *testing.T, xEnv *testenv.Environment) {
-		require.Eventually(t, func() bool {
-			xEnv.MakeGraphQLRequestOK(testenv.GraphQLRequest{
-				Query: `query { employees { id } }`,
-			})
-			logs := xEnv.Observer().FilterMessageSnippet("sanitize_utf8").All()
-			return len(logs) > 0
-		}, 10*time.Second, 500*time.Millisecond)
-
-		logs := xEnv.Observer().FilterMessageSnippet("sanitize_utf8").All()
-		require.NotEmpty(t, logs)
-		require.Equal(t, "otel error: traces export: string field contains invalid UTF-8: Enable 'telemetry.tracing.sanitize_utf8.enabled' in your config to sanitize invalid UTF-8 attributes.", logs[0].Message)
+		expected := "otel error: traces export: string field contains invalid UTF-8: Enable 'telemetry.tracing.sanitize_utf8.enabled' in your config to sanitize invalid UTF-8 attributes."
+
+		xEnv.MakeGraphQLRequestOK(testenv.GraphQLRequest{
+			Query: `query { employees { id } }`,
+		})
+
+		require.Eventually(t, func() bool {
+			logs := xEnv.Observer().FilterMessageSnippet("sanitize_utf8").All()
+			for _, entry := range logs {
+				if entry.Message == expected {
+					return true
+				}
+			}
+			return false
+		}, 10*time.Second, 500*time.Millisecond)
 	})
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@router-tests/telemetry/attribute_processor_test.go` around lines 240 - 250,
The test currently re-sends the GraphQL request on every Eventually poll and
then asserts logs[0], which causes flakiness; instead call
xEnv.MakeGraphQLRequestOK(...) once before polling, then use require.Eventually
to repeatedly read xEnv.Observer().FilterMessageSnippet("sanitize_utf8").All()
and check that any entry's Message equals the expected string (or that the slice
contains the expected message), finally assert the logs slice is not empty and
contains the expected message — update references in the test to
xEnv.MakeGraphQLRequestOK, xEnv.Observer().FilterMessageSnippet(...).All(), and
the expected message string so ordering of logs[0] is not relied upon.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@router-tests/telemetry/attribute_processor_test.go`:
- Around line 240-250: The test currently re-sends the GraphQL request on every
Eventually poll and then asserts logs[0], which causes flakiness; instead call
xEnv.MakeGraphQLRequestOK(...) once before polling, then use require.Eventually
to repeatedly read xEnv.Observer().FilterMessageSnippet("sanitize_utf8").All()
and check that any entry's Message equals the expected string (or that the slice
contains the expected message), finally assert the logs slice is not empty and
contains the expected message — update references in the test to
xEnv.MakeGraphQLRequestOK, xEnv.Observer().FilterMessageSnippet(...).All(), and
the expected message string so ordering of logs[0] is not relied upon.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: b5bde0d8-c13b-421e-8f3a-a4053ecf7eb2

📥 Commits

Reviewing files that changed from the base of the PR and between cfa19f7 and 03cf2da.

📒 Files selected for processing (1)
  • router-tests/telemetry/attribute_processor_test.go

@SkArchon SkArchon marked this pull request as ready for review March 6, 2026 10:16
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@router-tests/telemetry/attribute_processor_test.go`:
- Around line 241-251: The test currently calls xEnv.MakeGraphQLRequestOK inside
require.Eventually which replays the request each poll and produces multiple
logs; move the single call to xEnv.MakeGraphQLRequestOK out of the
require.Eventually so Eventually only polls for the log presence. Inside the
polling closure use xEnv.Observer().FilterMessageSnippet("sanitize_utf8").All()
to detect that any matching log has appeared (no request inside), and when
asserting the final log verify by searching the returned logs for an entry whose
Message equals the expected string instead of assuming logs[0] (use the Message
field comparison against "otel error: traces export: string field contains
invalid UTF-8: Enable 'telemetry.tracing.sanitize_utf8.enabled' in your config
to sanitize invalid UTF-8 attributes.").

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 37f0cb0a-86dc-446e-92bf-50e9bd8eed66

📥 Commits

Reviewing files that changed from the base of the PR and between 03cf2da and c175fee.

📒 Files selected for processing (1)
  • router-tests/telemetry/attribute_processor_test.go

Comment thread router-tests/telemetry/attribute_processor_test.go
@SkArchon SkArchon marked this pull request as draft March 6, 2026 11:19
@github-actions github-actions Bot added the router label Mar 6, 2026
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (1)
router-tests/telemetry/attribute_processor_test.go (1)

241-248: ⚠️ Potential issue | 🟠 Major

Avoid order-dependent assertion on logs[0] in a flaky-test fix path.

Line 248 assumes deterministic ordering of matching logs. If multiple sanitize_utf8 logs exist, this can still fail intermittently.

💡 Suggested stabilization
-			require.Eventually(t, func() bool {
-				logs := xEnv.Observer().FilterMessageSnippet("sanitize_utf8").All()
-				return len(logs) > 0
-			}, 10*time.Second, 500*time.Millisecond)
-
-			logs := xEnv.Observer().FilterMessageSnippet("sanitize_utf8").All()
-			require.NotEmpty(t, logs)
-			require.Equal(t, "otel error: traces export: string field contains invalid UTF-8: Enable 'telemetry.tracing.sanitize_utf8.enabled' in your config to sanitize invalid UTF-8 attributes.", logs[0].Message)
+			expected := "otel error: traces export: string field contains invalid UTF-8: Enable 'telemetry.tracing.sanitize_utf8.enabled' in your config to sanitize invalid UTF-8 attributes."
+			require.Eventually(t, func() bool {
+				logs := xEnv.Observer().FilterMessageSnippet("sanitize_utf8").All()
+				for _, log := range logs {
+					if log.Message == expected {
+						return true
+					}
+				}
+				return false
+			}, 10*time.Second, 500*time.Millisecond)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@router-tests/telemetry/attribute_processor_test.go` around lines 241 - 248,
The test is brittle because it asserts logs[0].Message equals the expected
string which fails if multiple matching "sanitize_utf8" entries appear; update
the assertion in attribute_processor_test.go to be order-independent by checking
that at least one entry in
xEnv.Observer().FilterMessageSnippet("sanitize_utf8").All() has Message equal to
the expected text (or assert the slice of messages contains the expected string)
instead of directly indexing logs[0]; locate the require.NotEmpty(t, logs) and
replace the require.Equal on logs[0].Message with a search/contains assertion
over the messages returned by FilterMessageSnippet.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@router-tests/telemetry/attribute_processor_test.go`:
- Around line 237-240: The test only checks that MakeGraphQLRequest returned no
error (require.NoError) but ignores the actual HTTP response, allowing non-200
responses to pass; capture the response value returned by
xEnv.MakeGraphQLRequest (e.g., resp, err := xEnv.MakeGraphQLRequest(...)) and
add an assertion that the response status is 200 (use require.Equal or
require.EqualValues against resp.StatusCode or resp.Status depending on the
response type) immediately after require.NoError to ensure non-200 responses
fail the test.

---

Duplicate comments:
In `@router-tests/telemetry/attribute_processor_test.go`:
- Around line 241-248: The test is brittle because it asserts logs[0].Message
equals the expected string which fails if multiple matching "sanitize_utf8"
entries appear; update the assertion in attribute_processor_test.go to be
order-independent by checking that at least one entry in
xEnv.Observer().FilterMessageSnippet("sanitize_utf8").All() has Message equal to
the expected text (or assert the slice of messages contains the expected string)
instead of directly indexing logs[0]; locate the require.NotEmpty(t, logs) and
replace the require.Equal on logs[0].Message with a search/contains assertion
over the messages returned by FilterMessageSnippet.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 39857a05-b4ac-433a-8916-6250f03dee87

📥 Commits

Reviewing files that changed from the base of the PR and between c175fee and bb9517f.

📒 Files selected for processing (3)
  • router-tests/telemetry/attribute_processor_test.go
  • router/pkg/trace/meter.go
  • router/pkg/trace/syncspan_test_exporter.go

Comment thread router-tests/telemetry/attribute_processor_test.go
@SkArchon SkArchon marked this pull request as ready for review March 6, 2026 12:00
@StarpTech
Copy link
Copy Markdown
Contributor

Thank you. It was already addressed in 24ebf0a

@StarpTech StarpTech closed this Mar 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants